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INTRODUCTION 



This document entitled EVALUATING TEACHER INSERVICE WORKSHOPS THROUGH 
A FOUR LEVEL ANALYSIS OF LEARNER BENEFITS offers the busy Workshop. Director 
an accurate and simple way to keep score of payoff to students taught by 
workshop participants. 

This publication responds to the following requests of Workshop 
Directors; 



We want clear-cut guidelines and directions on how to 
zero m on effective measures of our teacher inservice 
workshops. 

We want yardsticks with which to measure ourselves with 
other similar workshops. 

We want to find out the best type of format and the best 
type of workshop for the objectives we are attempting 
to achieve. 

The above requests have been put in simple basic English, it i 
possible to conie up with a more technical and scientific request: 

We want a comprehensive listing and categorization by 
type and level of claimed learner benefits directly 
linked to teacher inservice education. 

We want comparable data. 

We want comparable forms. 

We want comparable learner benefits. 

We want the workshop to start and stop as planned. 

We want an independent evaluation. 

We want teacher to attend these inservice workshops 
^o^* ^he competency being offered as well as for the 

r-rnr^' academic credits. ^ 

CI\1L O 



From both the common sense point of view and the scientific point of 
view, it would be difficult to summarize the objectives of this document; 
any clearer than done above • This document tries to meet the above requests. 

It must be noticed that certain of the above requests are general and 
can be answered in simple, non-technical English, These answers will be 
found by reading pages 1-8 which explain in detail the four lavels of 
learner benefits. 

There are many other technical questions that can be answered. For 
example, an ovetview cf the research undertaken to determine learner 
benefits ^nd possible levels of learner benefits is summarized on page 9. 

An analysis of the previous research in this area can be found on 
pages 10-11, 

The major purposes of this research in evaluating teacher inser\7ice 
workshops are listed on page 12, This research was conducted in three 
phases, which are explained in detail on pages 13-17^ 

In addition to a general description, each phase is described by 
title and sample products on pages 18-20, 

An overall timeframe is provided together with a critical path analysis 

y 

on page 21, 

For the statisician and researcher interested in scientific experimental 

research design, page 22 offers a concise analysis, 

A one page overview of this evaluation design is provided on page 24, 
The steps for developing the research instrument are described on 

pages 25-26, 

The distinction between the two major versions of the research instrument 
are explained on pages 27-29, 



Important procedures and forms necessary to conduct the survey are 
given on pages 30-31. 

In order to make sure that more than one method of gathering 
Evaluation data is encouraged, a number of evaluation possibilities have 
been listed on page 33. 

In order to specify the difference between the evaluation compenent 
performed by the Workshop Director and by the outside evaluator, page 34 lists 
three functions for both the Workshop Director and the outside evaluator. 

Since each of the functions of the Workshop Director and the outside 
evaluator will result formats and data, pages 35-39 give sample formats 
for each data function. 

Several agencies have begun to apply this evaluation procedure in a 
highly simplified, fashion. In order to encourage further applications, 
page 40 provides a short one-page overview of how this evaluation is being 
applied elsewhere. 

Applying this procedure elsewhere requires the development of evaluation 
dimensions. Each evaluation dimension is a specific component of a workshop 
or seminar to be evaluated. Pages 41 and 42 show how such an evaluation 
process can result in greater specification and clarity of these dimensions 
being evaluated. 
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A COMPREHENSIVE LISTING AND CATEGORIZATION 
BY TYPE AND LEVEL OF CL&DED LEARNER GAINS DIRECTLY 
LINKED TO TEACI^ INSERVICE EDUCATION 

During the period, September 1, 1973 to August 31, 1974, several 
teacher inservice education workshops were conducted* Each of these 
workshops had a list of objectives and anticipated outcomes o Some of 
these objectives and outcomes were in terms of teacher competencyo Other 
outcomes and objectives were in terms of benefits to learners o 

Since the ultimate purpose of prcrviding teacher benefits is to provide 
learner benefits, this listing of benefits is intended to provide a baseline 
data source against which any inservice workshop can compare itself o 

It would be an oversimplification to claim that any workshop objective 
or benefit presented in terms of teacher competency excluded learner gains ♦ 
Obviously, the teacher's competency is improved in order to benefit learners. 

The distinction made between a teacher benefit and a learner benefit 
refers to the explicitness of the prestated objectives of a specific workshop* 

In order to systematize this listing, the results are categorized by 
both type and level* 

Various types of benefits are recognizable. For example ^ knowledge 
benefits would include knowledge, information, data, principles, generalization^, 
and problem solving* Performance benefits would include performances, 
training, techniques, skills, procedures, and logistics^ Attitude benefits 
would include attitudes, motivation, values, emotions, htmian relations, 
priorities, and preferences^ 



TYPES OF BENEFITS 



Knowledge Benefits 
(DATA) 

Knowledge 

Information 

Data 

Principles 
Generalizations 
Problem Solving 
Thoughts 

Cognitive Behavior 



Performance Benefits 
(THINGS) 

Perf orinance 

Skill 

Things 

Procedures 

Routines 

Logistics 

Actions 

Psychomotor Behavior 



Attitude Benefits 
(PEOPLE) 

Attitude 

Values 

People 

Motivation 

Human Relations 

Priorities 

Emotions 

Affective Behavior 



In occupational literature, such as the DICTIONARY OF OCCUPATIONAL 
TITLES, it is usual practice to refer to DATA, PEOPLE, and THINGS. With 
this framework in mind, it is helpful to remember that normally speaking, 
knowledge benefits refer to DATA, performance benefits refer to THINGS, and 
attitude benefits refer to PEOPLE c This simplification should concretize 
each type of benefit. 

After the above analysis of type of benefit, it is in order to ask 
the following question, "How explicitly do prespecified objectives refer 
to student benefits?" 

Looking at the following four claims will help the reader conceptualize 
four different levels of student benefits 



I: Teachers will acquire competency A and ^ 
competency B as a result of inservice 
training. 

II: Teachers trained during inservice workshops 
^^^1 exhibit the newly acquired competency A 
and competency B with increased frequency 
and increased quality in teaching and 
educational influence upon learners. 

Ill: Teachers who have acquired competency A 

and competency B during inservice workshops 
will adapt such newly acquired competencies 
in order to remove environmental bc^rriers 
that interfer with the success with a 
newly acquired competency. 

IV: Teachers who have acquired competency A 

and competency B during inservice workshops 
will be able to provide evidence and documentation 
that link up the newly acquired conrpeteiicies 
with increased levels of learner success. 
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As Is obvious from overviewing these fotir levels o£ competency. 



LEVEL 



Improves the teacher as a person acquiring a new 
competency o 



UirVEL II 



Improves the teach-^r as a teacher exercising a 
new competency in a classroom situationo 



LEVEL III 



Improves the teacher functioning in a specific 
school with individualized local constraints© 



LEVEL IV 



Improves the documentation or evidence that 
individual learners are benefiting as a result 
of increased levels of teacher competency directly 
linked with inservice teacher workshops* 



It is important not to become entangled in semantics o In other words, 
the verbal content of a prestated objective helps determine the level of 
the objective. Similarly, the verbal content of an evaluation item helps 
determine the level of the evaluation item* Obviously, many prestated 
objectives of which the verbal content is on level I implicitly intend to 
achieve documented evidence of benefits to learners ♦ The reason for 
categorizing a specific objective or evaluation item on level I is the 
explicit verbal content of the statement* 

A few examples may be in order c The objective, "The teacher will be 
able to use slides during lecture presentations," is an example of a level I 
objective • The evaluation item, "Are you able as a result of this workshop 
to use slides during a lectinre presentation?" is an example of a level II 
evaluation item. Neither the objeccive nor the correlated evalxiation item 
asks about how frequently or X7ith what quality a teacher actually used 
slides during a lecture presentation; this would be a level II inquiry* 
Neither the objective nor the correlated evaluation item asks whettier or 
not the teacher adapted the slides used to specific local environments, school 



policies, student characteristics, subject matter, or topical interest; this 
would be a level III inquiry. Neither the objective nor the correlated 
evaluation item inquired about any evlv'ence or documentation that using 
such slides actually increased the learner gains of students exposed to this 
new teacher competency; this would be a level IV inquiry^ 

An example of a level II objpxtlve would be, "The teacher will use 
appropriate slides to accompany lecture presentations with greater frequency 
and with reasonable quality in the classroom." Obviously, the words appropriate 
and reasonable are subject to clarification. Nevertheless, the level of 
this objective is level II. The appropriate evaluation item would be, 
"Did this teacher in fact use slides in lecture presentations with greater 
frequency than last year and with observable improvement in the quality of 
the presentation?" This stress upon teacher as teacher is typical of 
level II. 

An example or a level III objective would be, "The teacher vill use 
slides during a lecture presentation in such a way as to make maximum utilization 
of available equipment, available projection rooms, an'^ available technicians •" 
This objective on level III may be imperfect whe*! examined fromthe point of 
view of conc'itions, perforroancej and criteria. However, it does show an 
attempt to help a teacher overcome difficult or different barriers that 
vary from school to school. An exan5>le of a correlated evaluation item on 
level III would be, "Did the teacher in fact use existing equipment with 
existing projection space, and with existing personnel?" Another example 
of an evaluation item would be, "Did the teacher try to use 35mm slides 
when the only type of projector available was 16mm?" If the teacher did not 
correlate the available slides with the available equipment, the teacher is 
obviously unsuccessful on level III. 
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An example of a level IV objective would be, "The teacher will be able 
to document learner gains on specific objectives through increased learner 
scores on objective multiple choice tests, on performance evaluation 
instnraients, and on attitude scales*" Even if a specific multiple choice 
type question item, a specific performance evaluation instrument, or a 
specific item on an attitude evaluation questionnaire were called into 
question, this attempt at docu^^antation meets the requirements of being 
on level IV# The correlated evaluation item could be, ^^Did the teacher 
provide some type of e/idence of learner gains?" 

This detailed explanation of four level analysis is intended to help 
readers conceptualize the following listing and categorization of claimed 
learner gains directly linked to teacher inservice education* 

The informed reader is aware of the fact that certain inservice education 
workshops will explicitly spell out benefits in terms of teacher competencies* 
These claims vill be subjected to TYPE ANALYSIS (for example, knowledge 
benefits, performance benefits, or attitude benefits) and to LEVEL AMLYSIS 
(level I, lev^l II, level III, level IV). 

After these claimed benefits have been analyzed, the next step would 
be to find out by sampling students whether or not any direct benefits were 
achieved as measured by student gains* 

It is not inconceivable that certain workshops with claima on level I 
achieve greater measurable student gains than other workshops with explicitly 
stated claims on level IV* 



On the other hand, it might be found that workshops in which the 
prestated claimed benefits were explicitly on level IV ref^ulted in greater 
measurable learner benefits ♦ 

One design for measiiring learner gains is to categorize learners into 
two groups 5 



GROUP A would include learners whose teachers have been 
trained during inservice education workshops, 

GROUP B would include students whose teachers have not 
been trained during inservice education workshops. 

There is a danger to this oversimplified approacho In order to avoid 
this danger, samples will be developed upon the following divisions: 

GROUP X would include students whose teachers possess 
competency A and competency Bo 

GROUP X-lwculd include students whose teachers have 
been trained during inservice education workshops, 

GROUP X-2wxild include students whose teacheiP have 
not been trained during inservice education programs, 

(Obviously, the teachers referred to in both groups X-1 and 
X-2 possess the competencies A and B listed aboveo) 

GROUP Y would include students whose teachers do not 
possess competency A anu competency B. 

Looking at tho above experimental design enables the questionnaire given 
to students to answer a number of specific questions: 

Is there a difference between the success of students 
taught by a teachei who has attended an inservice 
program and a teacher who has not attended an inservice 
program when both teachers possess the same competency? 



Is there a difference between the success of students 
taught by teachers who possess a specific competency 
and the success of students taught by teachers who do 
not possess a specific competency? 



This type of experimental design enables these and similar questions 
to be answered with a relatively simple evaluation instrument o 



A RESEARCH DESIGN FOR EVALUATING INSERVICE EDUCATION 



OVERVIEW 

This research proposal is designed to 

CLASSIFY, 

VALIDATE, and 

ANALYZE THE COSTS 
of inservice teacher education workshops in terms of 
MEASURABLE and EFFECTIVE BENEFITS to OCCUPATIONAL 
STUDENTS following teacher participation in inservice 
courses . 



PREVIOUS RESEARCH 



An analysis has been made of inservice education 
Ti>is analysis has classified the design of evaluation 
£.tudies into four levels. Each level has its own type 
of evaluation question : 



LEVEL I What is the new competency that the 

teacher has acquired as a result of 
inservice training? (Or, has the 
TEACHER ACQUIRED this competency?) 

LEVEL II What is the frequency and qua I i ty 

with which inservice trained teachers 
have exhibited the newly acquired 
competency in teaching and educational 
influence upon learners? 



LEVEL ill What barr i ers must be removed in 

hostile and difficult environments 
in order to make sure that the newly 
acquired competency is able to 
foster learner success? 

LEVEL IV What ev i dence or documentat i on is 

available to link up the newly acquired 
competency with increased levels of 
learner success? 

With this four level classification in mind, 
it is possible to sum up the available literature by 
saying that most previous research has been on levels 
I and !!• Very little research is available on levels 
III and iV. 

This research intends to zero in on levels 
III and IV. 



MAJOR PURPOSES 



The purpose of this research is to classify, 
validate, and analyze the costs of inservice teacher 
education workshops in terms of measurable and effective 
benefits to occupational students following teacher 
participation in inservice courses. 

The inservice courses being evaluated in this 
design are sponsored by the New York State Education 
Department through the Bureau of Inservice Education. 

The benefits being classified, validated, and *»- 
analyzed by cost benefit are those claimed by institutions 
and individuals conducting the individual inservice 
courses in State sponsored workshops. 

Since research * app I i es scientific criteria both 
to asking and answering relevant questions, it is necessary 
to delineate the questions being asked. 

5 • ^ 



Phase One : 

LlSl' the claimed benefits of teacher inservice courses 
that result in learner gains. 

Phase Two ; 

VALIDATE the presence of these claimed results UJi 
experimental samples . 

Phase Three: 

PRICE the validated benefits in a comparative shopping 
tabulation contrasted with value analysis. 



The following questions each constitute an 
essential phase of the research design : 
PHASE ONE : What claims are made about benefits of 
teacher inservice courses that result 
in learner gains'? 
PHASE TWO : What evidence or documentation is 
available to validate the above 
claimed benefits? 
PHASE THREE : Can the validated claims of inservice 
education be. achieved with equal 
documented quality by alternative 
programs at less cost? 



Each of these research questions can be broken 
down into directly observable and measurable subquesti 
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PHASE ONE SUBQUESTIONS AND TIME LINE 



SUBQUESTION 

I -A What are the names of the 

benef i ts c I a i med by i nser'v i ce 
courses in terms of learner 
9a i ns? 



TIME LINE 

before March 10, 75: 
ana I yze and class! fy 
sample of 

courses and stated 
object i ves 



l-'B How are these claimed benefits 
defined and measured? 



1~C How can these claimed benefits 
be used to i mprove ex i st i ng and 
planned inservice courses? 



before April 14/ 75: 
■ cor re I ate each claim 

benefit with both 
! oca I eva I uat i on •fool 
find a research based 
countab I e measure 

before Sept 8, 75: 
prepare for pub I i - 
cation a document 



ent i 1 1 ed 

A COMPREHENSIVE LISTING AND CATEG0RIZ/\TlO)V 
OF CLAIMED LEARNER GAINS DIRECTLY LINKED 
TO TEACHER INSERVICE EDUCATION 



ERIC 
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PHASE TWO SUBQUESTIONS AND TIME LINE 



SUBQUESTION 



TIME LINE 



2~A What sampi es can be used to provide 
objective mea.;ures of the impact of 
tnservice courses on learner gains? 



2-B Whdt d i f f erences can be measured 

in the learner gains between 

STUDENTS WHOSE TEACHERS HAVE RECEIVED 
INSERVICE TRAINING IN THE SUMMER OF 75 

and 



before May 15/ 75: 

sample 4 comparable 

groups with a 

p reassessment 

I nstrument based 

upon results of 1-B 

before Nov 17, 75: 
resample the 4 
experimental and 
control groups 



STUDENTS WHOSE TEACHERS HAVE NOT RECEIVED 
INSERVICE TRAINING IN THE SUMMER OF 75? 



3-C Can the claimed benefits be validated before Feb 2, 76: 
in experimental samples? statistically 

analyze the data 
gathered i n 2-B 

before Apr* 5/ 76:prepare for publication a document 

entitled : A VALIDATION ANALYSIS 
OF CLAIMED AND DOCUMENTED LEARNER 
GAINS DIRECTLY LINKED TO TEACHER 
INSERVICE EDUCATION 



ERLC 
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PHASE THREE SUBQUESTIONS AND TIME LINE 



SUBQUESTION 

3-A Which student benefits have been 

claimed and attempted by more than 
one alternative? 



TIME LINE 

before May 5, 75: 
correlate each 
claimed benefit 
with alternative 
courses and count- 
ab I e measures 



3-B How much does each enumerated 
benefit cost from available 
workshop budgets? 



before Sept i 5/ 75: 
develop c cost 
benefit model to 
use in conjunction 
with results of 2-B 



3~C In how many different ways and 
at how many different costs can 
documented student benef i ts be 
ach i e\ ed in i nserv i ce e ducat ion? 



before July I, 76: 
prepare for publi- 
cation a document 
entitled : 



A COST EFFECTIVENESS ANALYSIS AND 
A VALUE ANALYSIS OF DOCUMENTED 
LEARNER GAINS DIRECTLY LINKED 
TO TEACHER INSERVICE EDUCATION 
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Phase One: 



LIST the claimed benefits of teacher inservice courses 
that result in learner gains. 



Produc t One Title : 

A comprehensive listing and categorization of claimed learner 
gains directly linked to teacher inservice education. 



Sampl e of Product One : 



The following learner gain bendits are claimed: 



Name 

1. Benefit A 

2. Benefit B 



Definition 

A is 

B is 



Example 
Thus, 



For example 



Source 

• Teacher trained 

, Occ. Ed. Director 
. Supervisor (ERIC) 
, Parents 

. Teacher Trainer (CIJE) 

• Outside Evaluators 



ERIC 



Phase Two : 

VALIDATE the presence of these claimed results in 
experimental samples • 

Product of Two Title : 

A validation analysis of claimed and documented learner gains 
directly linked to teacher inservice educati )n. 



Sample of Product Two : 

Sample 1 Sample 2 Sample 3 Sample A 
!• Benefit A Yes Yes No Yes 
■2» Benefit B No Ng Yes No 
3. Benefit C No No No No 



Phase Three : 

PRICE the validated benefits in a comparative shopping 
tabulation contrasted with value analysis. 

Product Three Title : 

A cost effectiveness analysis and a value analysis of 

documented learner gains directly linked to teacher 
inservice education. 

Sample of Product Three : 

Name ISE Cost Alternative 1 Alternative 2 



1. 



Benefit A 



$5/pupil 



$l/pupil 



$3/pupil 



2. 



Benefit B 



$15/pupil 



$25/pupil 



$35/pupil 



3. 



Benefit C 



$25/pupil 



$25/pupil 



$26/pupil 
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RESEARCH DESIGN 



GROUPS: The following groups will be surveyed: 
GROUP X: Experimental 

Group X (Teachers) : 

. Teachers selected for training through inservice education 
courses in the summer of 1975 

(who become) 

• Teachers trained through inservice education workshop courses 
in the summer of 1975 

Group X (Students) : 

• Students taught by group X teachers 
GROUP Y: Control 

Group Y (Teachers): 

. Teachers neither selected for nor trained through inservice 
education workshop courses in the summer of 1975 

.Group Y (Stuaents): 
. Students taught by group Y teachers 
This type of sampling presumes that all teachers, both those in group X 
and those in group Y, have necessary teacher competencies. 

This type of sampling tries to measure the difference made by inservice 
workshop courses on increasing the impact made by teacher competency on 
learner gains. 



SAMPLING TIMETABLE: The following sampling timetable will be used: 

APRIL 1975: Teachers in group X and group Y are surveyed with the 
Form A (Teacher) instrument. 

Students of teachers in both group X and group Y are 
surveyed with the Form A (Student) instrument. 

OCTOBER 1975: Half of the teachers in group X and half of the teachers 
in group Y are surveyed with the Form A (Teacher) 
instrument* 

The other half of group X teachers and the other half of 
group Y teachers are surveyed with the From B (Teacher) 
instrument. 
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Half of the group X students and half of the group Y 
students are surveyed with the Form A (Student) 
instrument. 

The other half of group X students and the other half 
of group Y students are surveyed with the Form B 
(Student) instrument. 

APRIL 1976: Teachers in group X and group Y are surveyed with the 
Form B (Teacher) instrument. 

Students in group X and group Y are surveyed with the 
Form B (Student) instrumei^t. 

ALTERNATE FORMS OF MEASUREMENT INSTRUMENTS: In order to counteract the 
effect of practice, memorization, and evaluator bias, attempts have been made 
to develop both Form A and Form B. 

PHASE 1 : FORM A (Teacher and Student) is to be administered in 
April 1975. 

PHASE II : FORM A (Teacher and Student) and FORM B (Teacher and Student) 
are to be administered in October 1975. 

Form A (Teacher) is to be administered to half of the teachers both in 
group X and in group Y. Form B (Teacher) is to be administered to the other 
half, selected at random. 

The same randomization of Form A (Student) and Form B (Student) will be 
administered to both group X students and group Y students. 

PFIASE III : FORM B (Teacher and Student) is to be administered in April 

1976. 

This test-retest design helps measure the equivalence of Form A and 
Form B in assessing impact on learners. 

NOTE: If the items in the Form A instrument are such that the effects of 
practice, memorization, and evaluator bias are not statistically significant, 
later replication may merge Form A and Form B into Form AB. Form AB would 
then contain the nonduplicative items of Form A and Form B. 



AN EVALUATION DESIGN 
FOR 

TEACHER INSERVICE EDUCATION 
IN TERMS 
OF 

BENEFITS TO HIGH SCHOOL LEARNERS 



APRIL 1975 



SEPTEMBER 1975 



APRIL 1976 



Form A : Measure 

Learner- centered benefits 
based upon teacher-cct.tered 
objectives DEVELOPED BY 
WORKSHOP DIRECTOR 

. Does the teacher 

perform skill 1 as trained? 

. Yes / No 

• Frequency 

• Quality 

« Does the learner benefit? 



Form A: 



Form A: 



Same as April 75 
but with REVISIONS 
and ADDITIONS. 



Same as Sept. 75 
but with ADDITIONAL 
REVISIONS and 
ADDITIONS, 



Form B : In preparation 



Form B: Measure 



Form B: 



Learner-centered 
benefits based upon 
NYSED expectations 
DEVELOPED by Div. of 
Occ. Ed., BOER, and 
BISE. 



Same as Sept. 75 
but with REVISIONS 
and ADDITIONS. 



Part 1 

Applicable to all occupa- 
tional workshop areas 

BAE 

BBE 

BDE 

BHOE 

BHEE 

BITE 
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Part 2 

Applicable to one occupa- 
tional workshop area. 



3^ 



DEVELOPING FORM A 
FOR APRIL 1975 



STEP 1: 

Obtain clearance for use of sxitnmer 1975 objectives from BISE. 
STEP 2: 

Obtain copies of objectives for inservice workshops for summer of 
1975 from BISE. 

STEP 3: 

Analyze objectives from workshops for summer of 1975 according 
to the following criteria: 

Appropriate? 
Specific? 

Measurable or countable? 
Reasonable? 
Teacher competency? 
Learner benefit? 

STEP 4: 

Communicate the results of the analysis performed in Step 3 to each 
individual workshop director. 

A. For feedback 

B. For editing 

C. For revision 

D. For improvement 

STEP 5: 

Reanalyze the results of the revision performed by workshop 

directors in Step 4 according to the following criteria: 

A» Appropriate 

B. Specific 

C. Measurable or countable 

D. Reasonable 

E. Teacher competency 

F. Learner benefit 




STEP 6: 



Translate the results of the preceding five steps into 
form A evaluation items under two general headings: 

A. Teacher-centered objectives 

B. Learner-centered benefits 

STEP 7: 

Edit the draft version of form A in consultation with: 

BISE staff 
BOER chief 

Appropriate occupational education 

bureau chief 
Division of Occupational Education 
Division of Occupational Education Supervision 

STEP 8: 

Conduct the planned April 1975 sampling 



NOTE: Since it can be anticipated that some of the inservice 
workshop directors will not cooperate with Step 4 in 
sufficient time for the April sampling, a MINIMUM of 
five inservice workshops will be completely processed 
as specified in steps 1 through 7. 

Since BISE requires of each inservice workshop director 
to provide some type of preassessment in order to 
pretest the participating teachers, steps will be 
taken to syscematize this data available from BISE 
as u parallel and equivalent sample to the more 
formal FOflM A. 



THE DISTINCTION BETWEEN FORM A AND FORM B 



Form A is derived basically from workshop objectives as developed by 
the inservice workshop director. Form A is edited and revised by State 
Education personnel, but the basic data source is the workshop director. 

FORM B is basically derived from expectations, criteria, regulations, 
and concerns derived basically from State Education personnel* 

As such. Form B will be concerned with certain findings that are 
applicable to ALL occupational workshop areas as well as with findings 
that are applicable ONLY TO ONE occupational workshop area. 

The obvious CAVEAT hare is the fact that data collected can only be 
as valuable as the instrument with which the data ^s collected* This means 
that both Form A and Form B must be kept as simple as possible in order to 
facilitate validation*, 

This means operationally that both Form A and Form B should be: 

1* Easy to comprehend and conceptualize 

2. Easy for participants to fill out 

3. Easy for the workshop director and 
State Education personnel to simimarise 
and utilize 

STATISTICAL IMPLICATIONS 

There are many different ways to analyze data for si^jnlf icance 
of the difference in order to cpeed up the availability of the data. 
The following steps will be used. 

1. The mean will be determined for both 
group X and group Y. 



2. The standard deviation will be determined for 
both group X and group Y. 

3. The standard deviation of the difference 
between two means will be determined. 

4. The difference between the two means 
will be determined. 

5. The difference between the two means will 
be divided by the standard deviation of 
the difference. 

6. The level of significance will be specified 
at the 1%, 57o, and 10% level of significance. 

7. Differences not significant at least the 1""% 
level of confidence will be marked NS or no*, 
significant. 

8. On the original evaluation instrument, 
immediately next to each evaluation, two 
columns will be appended giving the results 
of significant tests for both group X and 
group Y. 



SAMPLE DATA ANALYSIS 



Look at the following sample data analysis. 

The first column identifies threo different evaluation items. Each 
of these evaluation items is a different question on Form A. 

The second column reveals the results for group X. Group X refers 
either to teachers trained in inservice education workshops or to 
students taught by teachers trained in inservice education workshops. 

The third column gives the results for group Y. Group Y refers 
to teachers not trained in inservice education workshops and to students 
taught be teachers not trained in inservice education workshops. 
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GROUP X 



GROUP Y 



ITEM 1 



57. 



ITEM 2 



ITEM 3 



NS 



Looking at the above data shows three different possible results of 
the April 1975 testing. ^ s 

It must be noted that during the April 1975 testing, group X refers 
to teachers that are going to participate in inservicp education workshops 
during the summer of 1975. 

ITEM 1 gives the situation wherein on one specific test xcem, group X 
was superior to group Y at the 5% level of confidence. If the vast 
majority of test items come out with' group X superior at the start of the 
measurement, then obviously the experimentation is stacked in favor of 
group Y. 

ITEM 2 gives an example wherein, on one specific test item, group Y 
was obviously superior to group X. 

ITEM 3 gives the desirable situation wherein, at the beginning of 
the experimentation, both group X and group Y are equivalent. Equivalent 
means that on the variables under study, there is n'o statistically 
significant difference between group X and group Y. 
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Conducting the Survey 

Select BOCES (and other eligible districts) 

A. BOCES with applicants 

1) BOCES with applicants accepted 

2) BOCES with applicants rejected 

B. BOCES with no applicants 



INSERVICE BENEFITS, NEEDS, AND PRIORITY ASSESSMEOTS 
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Claimed Overall Inservice 
Benefits 

(Summer 1975) 


% of teachers who 
need this benefit 


% of teachers who 
already possess 
this competencv 


% of teachers for 
whom this benefit 
is NOT APPLICABLE 


% of students who 
1 need this benefit 


% of students who 
already possess 
this competencv 


% of students for 
1 whom this benefit 
is NOT APPLICABLE 


Prioritv Rank 

1 = Highest priority 

Start with 1 and work 
down to rank all 
benefits. 


A. DATA: Participants will up- 

date themselves with 
the latest trends and 
information, 

B. PEOPLE: Participants will ex- 

change with one 
another a wide variety 
of unique and indivi- 
dualizing approacheii 

C. THINGS: Participants will 

acquire new and needed 
hands-on skills that 
reflect the best 
available occupational 
practices 














A 














jj 














c 

Rank A, B and C witb 
1, 2 and 3 


D. OCCUPATIONAL AREA: 

Update participants in the 
area of: 

D-1 Agriculture Education 
D-2 Business Education 
D-3 Distributive Education 
D-4 Health Occupations Ed. 
D-5 Home Economics Education 
D-6 Trade & Technical Ed. 














D-1 














D-2 














D-3 














D-A 














D-5 














U -0 

ivaniv u — L cu u "O wicn 
1, 2. 3, 4, 5 and 6 


E. WORKSHOP TOPIC: 

Topic 1: 
Topic 2: 

• 

• 

• 

Topic N 














Topic 1 














Topic 2 














• 














Topic N 

Rank workshop topics 1 
to N with 1, 2, 3. . . .n 


ERLC 


Please round off the above 
percentages to the nearest^^^ 
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WORKSHOPS, OCCUPATIONAL BUREAU, AND SPECIALTY 





Workshop 1 

Area: BAE 
Horticulture 


Workshop 2 

Area: BBE 
Bookkeeping 


Workshop 3 

Area: BDE 
Marketing 


Workshop 4 

Area: BHOE 
Trends 


Workshop 5 

Area: BHEE 
Modules 


Workshop 6 

Area: BTTE 
Welding 


• • • 


Workshop x 

Area: Bxxx 
Specialty 


TOTAL Per 
BOCES 


Number of Eligibles, Applicants, Accepts Per BOCES j 


BOCES A 




















BOCES B 




















BOCES C 




















BOCES D 




















BOCES E 




















BOCES F 




















BOCES G 




















BOCES H 




















BOCES I 




















• 
• 
• 




















BOCES N 
















x-y-z 


^BOCES 


Total Per 
Occupational 
Specialty 
Area 
















^Spec. 





SAMPLE EVALUATION POSSIBILITIES 



I nterna I 
Externa I 

Survey 

Quest i onna i re 
Teacher eva I uat i on 
Check I i st 
Observat i on 
Anecdota I 
Rat i ng sea I es 
State forms 
Psycho I eg i st 

Ach i evement pretest i ng 
Ach i evement posttest i ng 
Behavioral change 

Count : 

# time^ service used 

# part i c i pants 

# us i ng serv i ces 

# service units 

# phone i nqu i r i es 
jt hours of service 

# spontaneous pl aud i ti 
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INSERVIC E WORKSHOPS EVALUATION DESIGN 
Spring - Summer 1975 



Workshop Director 

does the following: 

Develops purposes, goals, objec- 
tives, benefits , processes, plans , 
products, or reasons for the 
workshop in concert with staff. 



BOER Evaluator 

does the following: 

1^ Analyzes workshop expectations to 
see if 

a. appropriate to inservice education 
b^ specific 

c. measurable or countable 

d. reasonable 



2. Approves an individualized evalu- 
ation instrument developed, 
revised, or selected by workshop 
staff as an acceptable yardstick 
of success (Le., measurable gains). 



2. Correlates : 

a. each evaluation item with the 
prespecified expectations of the 
workshop 

b. each workshop expectation and 
evaluation item with a measurable 
learner benefit acceptable to the 
workshop director 



3. Surveys : 

a. each staff member and partici- 
pant with the approved evaluation 
instrument at least at the end of. 
the workshop 

b. administrators, teachers, and 
students to measure impact upon 
classroom instruction and learning 



3. Samples independently in October 1975 
following workshop 

a. staff members and director 

b. supervisors or administrators 

c. workshop participants (classroom 
teachers) 

d. non-participants (classroom 
teachers) 

e. students (of participants) 

f. students (of non-participants) 



DEVELOP WORKSHOP PLANS 



Benefit A 
Benefit B 
Benefit C 
Benefit D 
Benefit E 



APPROVE EVALUATION INSTRUMENT 

Item 1 _______ 

2 - 

3 

4 

5 

6 

7 

8 

9 

10 

11 

12 
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ANALYZE BENEFITS 





Appropriate 


Specific 


Measurable or 
Countable 


Reasonable 




9 


Nn 


9 


Via o 

xes 


BenGflt B 


Yes 






les 


Benefit C 


Yes 




No 


? 


Benefit D 


No 


Yes 


Yes 


No 


Benefit E 


Yes 


Yes 


Yes 


No 
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CORRELATE EVALUATION ITEMS AND BENEFITS 



Benefit A is, measured by : 

Item 10 

Item 11 

Benefit B measured by : 

Item 1 

Item 5 

Item 9 

Benefit C i^ measured by : 

Item 2 

Benefit D is measured by : 

Item 3 

Item 7 

Benefit E is^ measured by : 

Item 4 

Item 6 

Item 8 

Item 12 
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Letter Code 



la 



lb 



CORRELATE SAMPLING INSTRUMENT WITH 
WORKSHOP EVALUATION INSTRUi^ENT 

CODING 



Description 



Teacher performs 

Teacher is observed 
performing 

Learner performs 



Evaluation Item 



Did you as a teacher perform 
competency 1? 

Did the teacher being observed 
perform competency 1? 

Did you as a student acquire 
success 1? 



Ic 



Learner is observed 
performing 



Did your students acquire 
success 1? 



^ 

Workshop 
Instrument 


Workshop 
Partici- 
pant 


FOLLOWUP PARALLEL SAMPLES 


Dir. 


Staff 


Sugvrs 
Admxn. 


Partici- 
pants 


Non- 
Part. 


Stud, of 
Part. 


Stud, of 
Non-part. 


Item 1 


1 


la 


la 


la 


1 


1 


lb 


lb 


Item 2 


2 


2c 


2c 


2c 


2c 


2c 


2b 


2b 


Item 3 


3 


3a 


3a 


3a 


3 


3 


3a 


3a 


Item 4 


4 


4c 


4c 


4c 


4c 


4c 


4a 


4a 


Item 5 


5 


5a 


5a 


5a 


5 


5 


5b 


5b 


Item 6 


6 


6c 


6c 


6c 


6c 


6c 


6a 


6a 


Item 7 


7 


7a 


7a 


7a 


7 


7 


7a 


7a 


Item 8 


8 


8c 


8c 


8c • 


8c 


8c 


8a 


8a 


Item 9 


9 


9a 


9a 


9a 


9 


9 


9b 


9b 


Item 10 


10 


10a 


10a 


10a 


10 


10a 


10b 


10b 


Item 11 


11 


11a 


11a 


11a 


11 


11a 


lib 


lib 


Item 12 


12 


12c 


12c 


12c 


12c 


12c 


12b 


12b 
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SAMPLE RESULTS 



Benefit 
Claimed 


Evaluation 


Wkshop 
Dir. 


Wkshop 
Staff 


Supvrs 
Admin. 


Parti- 
cipants 


Non- 
Part. 


Stud, of 
Part. 


Stud, of 
Non-part . 


A 


Item 10 


100% 


75% 


60% 


50% 


40% 


30% 


31% 




Item 11 


0 


0 


40 


50 


10 


70 


10 


B 


Item 1 


100 


80 


10 


10 


30 


15 


30 




Item 5 


100 


80 


80 


70 


30 


60 


20 




Item 9 


100 


80 


85 


75 


25 


65 


25 


C 


Item 2 


100 


50 


55 


55 


40 


52 


51 


D 


Item 3 


100 


50 


40 


30 


30 


15 


5 




Item 7 


0 


25 


30 


20 


15 


20 


31 


E 


Item 4 


0 


0 


10 


20 


10 


30 


10 




Item 6 


100 


80 


70 


80 


40 


70 


30 




Item 8 


0 


60 


70 


90 


30 


80 


20 




Item 12 


100 


80 


70 


80 


40 


70 


20 



ANALYSIS 




Benefit A = 


9 


Benefit B = 


Yes 


Benefit C = 


9 


Benefit D = 


No 


Benefit E = 


Yes 



ERIC 
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BTT EVALUATION MODEL 

For each workshop, three evaluation forms will be developed: 

Pre-assessment form 

End of session evaluation form 

Pos t -a s s es smen t form 

For each of these forms, a review will be conducted by BTT and BOER 
to determine the need for supplemental questions, if any. 

For each of these forms, a certain overlap is expected from the fact 
that specific questions may appear on all three forms. This overlap is 
really a sequential followup in order to pinpoint progress. 

From a systematic point of view, it is hoped that written evaluation 
instruments will suffice to elicit the proper data from participants 
performing the evaluation. 

In order to verify this assumption, data collected from on site 
evaluators will be compared with the data gathered through written evaluation 
instruments . 



DRAFT 

DISCUSSION OUTLINE ONLY 

WORKSHOP CHARACTERISTICS 



The new format of workshops will consist of having five one-day sessions 
spaced throughout the school year and conducted by colleges with teachers who 
have been previously exposed to the one-week workshop format. 

The above format can be subdivided into four characteristics: 

1. Five one-day workshops 

2. Spaced throughout the school year 

3. Conducted by colleges 

4. For teachers who have experienced the 
one-we<ik workshop format 

Each of these dimensions poses a certain number of evaluation questions: 



DIMENSION 1 : FIVE ONE-DAY WORKSHOPS 

A. Is this format more or less productive? 

B. In what way? 

C. What is the effect upon the continuity 
of content and attendance? 

DIMENSION 2 : SPACED THROUGHOUT THE SCHOOL YEAR 

A. Can better services be rendered during 

the school year or at the end of the school year? 

B. What are the administrative changes at the 
local level and at the State education level? 

C. What is the overall effect of such an 
arrangement on sending schools? 

DIMENSION 3 : CONDUCTED BY COLLEGES 

A. Do people who don't need credit attend these workshops? 

B. Why do people attend these workshops? 

C. Does the academic credit earned at these workshops 
count towards a salary increment? 

DIMENSION 4 : TEACHERS WHO HAVE ATTENDED WEEK LONG SUMMER SESSIONS 

A. Can the week long sunmier sessions be compared with 

the five one-day sessions spaced throughout the school year? 

B. For which objectives or purposes are the week long 
summer workshops more appropriate? 

C. For which objectives or purposes are the five one-day 
workshops spaced throughout the school year more 
appropriate? 

D. How do teachers perceive each of these two contrasting 
workshop formats? 
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DRAFT 2 
DISCUSSION OUTLINE ONLY 
Nov. 18, 1974 



WORKSHOP CHARACTERISTICS 



The new workshop format will be character iEed by the following dimensions: 
(1) five one-day sessions; (2) spaced throughout the school year; (3) conducted 
in colleges; (4) attended by teachers previously exposed to the one-week 
workshop format; and (5) offering college credit to workshop participants. 

Each of the above dimensions poses a certain number of evaluation 
questions : 



DIMENSION 1 ; FIVE ONE-DAY WORKSHOPS 
A. Is this format productive? 

In what way? 
C. What is the effect upon the continuity 

of content and attendance? 

DIMENSION 2 ; SPACED THROUGHOUT THE SCHOOL YEAR 

A. Can better services be rendered during the school 
• year or at the end of the school year? 

B. What are the administrative change implications of these 
workshops at the local level and at the State education level? 

C. What is the overall effect of such an arrangement on 
sending schools? 

DIMENSION 3 : CONDUCTED IN COLLEGES 

A. Do people who don't need credit attend these workshops? 

B. Why do people attend these workshops? 

C. Does the academic credit earned at these workshops 
count towards a salary. increment? 

DIMENSION 4 ; ATTENDED BY TEACHERS PREVIOUSLY EXPOSED TO ONE-WEEK 
WORKSHOP FORM/^T 

A. Can the week long summer sessions be compared with the 
five one-day sessions spaced throughout the school year? 

B. For which inservice education objectives or purposes are 
the week long summer workshops more appropriate? 

C. For which inservice education objectives or purposes are 
the five one-day workshops spaced throughout the school 
year more appropriate? 

D. How do teachers perceive each of these two contrasting 
workshop formats? 

DIMENSION 5 ; OFFERING COLLEGE CREDIT TO WORKSHOP PARTICIPANTS 

A. Does college credit increase the practical value of 
these workshops? 

B. How is graduate credit distinguished from undergraduate 
credit? 
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CONCLUSIONS 



This document has contained both general and technical explanations 
The general explanations are found on pages 1-8. 
The technical explanations are found on pages 9-42. 



As always, the result of reading such a document is the whetting 
the appetite of the reader. 



For those who want more information, send a request on letterhead t 



Howard P. Alvir, Ph.D 
Associate in Research 

Bureau of Occupational Education Research 
Room 468 EBA 

New York State Education Department 
Albany, NY 12234 



Be sure to include: 



A stamped (18^) 

self-addressed (8^x11) envelop 

marked SPECIAL FOURTH CLASS RATE: BOOKS 



For more information, write to: 



AUTHOR : Howard p. Alvir, Ph^D. 

Associate in Research 

Bureau of Occupational Education Research 
Room 468 EBA 

New York State Education Department 
Albany, New York 12234 



For microfiche and hard copies, write to: 

ERIC Document Reproduction Service 
P.O. Box 190 

Arlington, Virginia 22210 



For workshops, seminars, translations, and 
consultations on this topic, write to: 



Dr. Marcel Laval lee 
SPONSORING G.R.E.C. — F.I.L.M.S. 
AGENCY Sciences de I'Education 

NAME : U.Q.A.M. 

CP. 8888 

Montreal, P.Q., Canada 



Be sure to include: 
A stamped (18c) 

self -addressed (8^x11) envelope 

marked SPECIAL FOURTH CLASS RATE: BOOKS 



*Refer to ICIS document number , and also the 
complete title. 

For this document, the ICIS number is # 7001 



